{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_intro_DLDIY_colab.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "toc_visible": true,
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mlelarge/dataflowr/blob/master/CEA_EDF_INRIA/01_intro_DLDIY_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-SvfKxAGCdhF"
      },
      "source": [
        "# Using CNN for dogs vs cats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VI1VQlwECdhG"
      },
      "source": [
        "We are going to create a model to enter the [Dogs vs Cats](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition) competition at Kaggle."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dyhW08piCdhH"
      },
      "source": [
        "There are 25,000 labelled dog and cat photos available for training, and 12,500 in the test set that we have to try to label for this competition. According to the Kaggle web-site, when this competition was launched (end of 2013): *\"**State of the art**: The current literature suggests machine classifiers can score above 80% accuracy on this task\"*. So if you can beat 80%, then you will be at the cutting edge as of 2013!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "h4dAhx8-CdhM"
      },
      "source": [
        "##  Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6oU4Z_DPCdhM",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import os\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision\n",
        "from torchvision import models,transforms,datasets\n",
        "import time\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "A66_r51xCdhS",
        "colab": {}
      },
      "source": [
        "torch.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "KN3FFTFhHQyi",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "sys.version"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tuej9DPjCdhX"
      },
      "source": [
        "Check if GPU is available and if not change the [runtime](https://jovianlin.io/pytorch-with-gpu-in-google-colab/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "t56d0zbFCdhY",
        "colab": {}
      },
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "print('Using gpu: %s ' % torch.cuda.is_available())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dDLlOjT5Et4p"
      },
      "source": [
        "## Downloading the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FSmB5oKYCdhc"
      },
      "source": [
        "You can download the full dataset from Kaggle directly.\n",
        "\n",
        "Alternatively, Jeremy Howard provides a direct link to the catvsdogs [dataset](http://files.fast.ai/data/dogscats.zip). He's separated the cats and dogs into separate folders and created a validation folder as well. You'll need this folder structure to run VGG.\n",
        "\n",
        "For test purpose (or if you run on cpu), you should use the (small) sample directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Rnn5pLK6EyJK",
        "colab": {}
      },
      "source": [
        "%mkdir data\n",
        "%cd /content/data/\n",
        "!wget http://files.fast.ai/data/dogscats.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BTobJ9vTE37J",
        "colab": {}
      },
      "source": [
        "!unzip dogscats.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XsMtmnCbCdhd",
        "colab": {}
      },
      "source": [
        "%ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "S5OW01WzCdhu",
        "colab": {}
      },
      "source": [
        "%cd dogscats/\n",
        "%ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mkj3DZjpCdha"
      },
      "source": [
        "## Data processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iasXk_FKCdhy",
        "colab": {}
      },
      "source": [
        "%cd .."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MJRnJgGOCdh4",
        "colab": {}
      },
      "source": [
        "data_dir = '/content/data/dogscats'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "U3lE0cvyCdh8"
      },
      "source": [
        "```datasets``` is a class of the ```torchvision``` package (see [torchvision.datasets](http://pytorch.org/docs/master/torchvision/datasets.html)) and deals with data loading. It integrates a multi-threaded loader that fetches images from the disk, groups them in mini-batches and serves them continously to the GPU right after each _forward_/_backward_ pass through the network.\n",
        "\n",
        "Images needs a bit of preparation before passing them throught the network. They need to have all the same size $224\\times 224 \\times 3$ plus some extra formatting done below by the normalize transform (explained later)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8t4vokNrF19p",
        "colab": {}
      },
      "source": [
        "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
        "\n",
        "vgg_format = transforms.Compose([\n",
        "                transforms.CenterCrop(224),\n",
        "                transforms.ToTensor(),\n",
        "                normalize,\n",
        "            ])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "l8LMReVECdh-",
        "colab": {}
      },
      "source": [
        "dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), vgg_format)\n",
        "         for x in ['train', 'valid']}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pMh7kjEBCdiC",
        "colab": {}
      },
      "source": [
        "os.path.join(data_dir,'train')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VNyS0TyeKwdO"
      },
      "source": [
        "Interactive help on jupyter notebook thanks to `?`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WX5pkMX4CdiF",
        "colab": {}
      },
      "source": [
        "?datasets.ImageFolder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oYmIGNEjKwdR"
      },
      "source": [
        "We see that `datasets.ImageFolder` has attributes: classes, class_to_idx, imgs.\n",
        "\n",
        "Let see what they are?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "m9ifn_R7CdiH",
        "colab": {}
      },
      "source": [
        "dsets['train'].classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TB6sTwFuCdiK",
        "colab": {}
      },
      "source": [
        "dsets['train'].class_to_idx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "y9ECrD2ACdiO",
        "colab": {}
      },
      "source": [
        "dsets['train'].imgs[:5]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WefhjZb2CdiQ",
        "colab": {}
      },
      "source": [
        "dset_sizes = {x: len(dsets[x]) for x in ['train', 'valid']}\n",
        "dset_sizes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SCLV1YgaCdiT",
        "colab": {}
      },
      "source": [
        "dset_classes = dsets['train'].classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Zy52-XhACdiX"
      },
      "source": [
        "The ```torchvision``` packages allows complex pre-processing/transforms of the input data (_e.g._ normalization, cropping, flipping, jittering). A sequence of transforms can be grouped in a pipeline with the help of the ```torchvision.transforms.Compose``` function, see [torchvision.transforms](http://pytorch.org/docs/master/torchvision/transforms.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iphL57PhKwdh"
      },
      "source": [
        "The magic help `?` allows you to retrieve function you defined and forgot!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "X_ARYwraCdiY",
        "colab": {}
      },
      "source": [
        "?vgg_format"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tWnJQiWgGP_R",
        "colab": {}
      },
      "source": [
        "loader_train = torch.utils.data.DataLoader(dsets['train'], batch_size=64, shuffle=True, num_workers=6)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Iern_6GNCdie",
        "colab": {}
      },
      "source": [
        "?torch.utils.data.DataLoader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wN1BKHfDCdig",
        "colab": {}
      },
      "source": [
        "loader_valid = torch.utils.data.DataLoader(dsets['valid'], batch_size=5, shuffle=False, num_workers=6)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Z4Be7lSLCdik",
        "colab": {}
      },
      "source": [
        "count = 1\n",
        "for data in loader_valid:\n",
        "    print(count, end=',')\n",
        "    if count == 1:\n",
        "        inputs_try,labels_try = data\n",
        "    count +=1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3BvxQqfzCdiq",
        "colab": {}
      },
      "source": [
        "labels_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MLNsfqc8Cdis",
        "colab": {}
      },
      "source": [
        "inputs_try.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vsaL21ouKwd9"
      },
      "source": [
        "A small function to display images:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "346yl-gbcLLm",
        "colab": {}
      },
      "source": [
        "def imshow(inp, title=None):\n",
        "#   Imshow for Tensor.\n",
        "    inp = inp.numpy().transpose((1, 2, 0))\n",
        "    mean = np.array([0.485, 0.456, 0.406])\n",
        "    std = np.array([0.229, 0.224, 0.225])\n",
        "    inp = np.clip(std * inp + mean, 0,1)\n",
        "    plt.imshow(inp)\n",
        "    if title is not None:\n",
        "        plt.title(title)\n",
        "    plt.pause(0.001)  # pause a bit so that plots are updated"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IJbW-QzGCdiv",
        "colab": {}
      },
      "source": [
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs_try)\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8ZK7nRSWKweH"
      },
      "source": [
        "What is an image for your computer?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hrzxYMXXCdix",
        "colab": {}
      },
      "source": [
        "inputs_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2HvFNWJICdiz",
        "colab": {}
      },
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(loader_train))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iUrTGlTpCdi5",
        "colab": {}
      },
      "source": [
        "# Get a batch of validation data\n",
        "inputs, classes = next(iter(loader_valid))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LOvkYiROCdi7"
      },
      "source": [
        "## Creating VGG Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BO5LAG4bCdi7"
      },
      "source": [
        "The torchvision module comes with a zoo of popular CNN architectures which are already trained on [ImageNet](http://www.image-net.org/) (1.2M training images). When called the first time, if ```pretrained=True``` the model is fetched over the internet and downloaded to ```~/.torch/models```.\n",
        "For next calls, the model will be directly read from there."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "K9PsHjXgCdi9",
        "colab": {}
      },
      "source": [
        "model_vgg = models.vgg16(pretrained=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VBzGe2mlCdi-"
      },
      "source": [
        "We will first use VGG Model without any modification. In order to interpret the results, we need to import the 1000 ImageNet categories, available at: [https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json](https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qhvs0ki_Cdi_",
        "colab": {}
      },
      "source": [
        "!wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Hppljqo5CdjC",
        "colab": {}
      },
      "source": [
        "import json\n",
        "\n",
        "fpath = '/content/data/imagenet_class_index.json'\n",
        "\n",
        "with open(fpath) as f:\n",
        "    class_dict = json.load(f)\n",
        "dic_imagenet = [class_dict[str(i)][1] for i in range(len(class_dict))]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QATvAFILCdjF",
        "colab": {}
      },
      "source": [
        "dic_imagenet[:4]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "p6QQhwruCdjI",
        "colab": {}
      },
      "source": [
        "inputs_try , labels_try = inputs_try.to(device), labels_try.to(device)\n",
        "\n",
        "model_vgg = model_vgg.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "epMB0UF9CdjM",
        "colab": {}
      },
      "source": [
        "outputs_try = model_vgg(inputs_try)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dOlx7YcPCdjO",
        "colab": {}
      },
      "source": [
        "outputs_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "OrweFfK9k0TW",
        "colab": {}
      },
      "source": [
        "outputs_try.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nk538XVpHc0y",
        "colab_type": "text"
      },
      "source": [
        "To translate the outputs of the network into 'probabilities', we pass it through a [Softmax function](https://en.wikipedia.org/wiki/Softmax_function)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MCIxHN2QCdjT",
        "colab": {}
      },
      "source": [
        "m_softm = nn.Softmax(dim=1)\n",
        "probs = m_softm(outputs_try)\n",
        "vals_try,preds_try = torch.max(probs,dim=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RGTVkH2pk0Tb",
        "colab": {}
      },
      "source": [
        "torch.sum(probs,1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "CvGbb2bQCdjZ",
        "colab": {}
      },
      "source": [
        "vals_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0TUdNKCJCdjc",
        "colab": {}
      },
      "source": [
        "print([dic_imagenet[i] for i in preds_try.data])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jtzuHob9Cdjf",
        "colab": {}
      },
      "source": [
        "out = torchvision.utils.make_grid(inputs_try.data.cpu())\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try.data.cpu()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "f3Bktt5PCdjh"
      },
      "source": [
        "### Modifying the last layer and setting the gradient false to all layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "L8kr3-tjCdji",
        "colab": {}
      },
      "source": [
        "print(model_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IXoMW73MCdjl"
      },
      "source": [
        "We'll learn about what these different blocks do later in the course. For now, it's enough to know that:\n",
        "\n",
        "- Convolution layers are for finding small to medium size patterns in images -- analyzing the images locally\n",
        "- Dense (fully connected) layers are for combining patterns across an image -- analyzing the images globally\n",
        "- Pooling layers downsample -- in order to reduce image size and to improve invariance of learned features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hA2f5FRuCdjm"
      },
      "source": [
        "![vgg16](https://mlelarge.github.io/dataflowr/Notebooks/vgg16.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SK6lfAzfCdjn"
      },
      "source": [
        "In this practical example, our goal is to use the already trained model and just change the number of output classes. To this end we replace the last ```nn.Linear``` layer trained for 1000 classes to ones with 2 classes. In order to freeze the weights of the other layers during training, we set the field ```required_grad=False```. In this manner no gradient will be computed for them during backprop and hence no update in the weights. Only the weights for the 2 class layer will be updated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rQwRKKC-Cdjo",
        "colab": {}
      },
      "source": [
        "for param in model_vgg.parameters():\n",
        "    param.requires_grad = False\n",
        "model_vgg.classifier._modules['6'] = nn.Linear(4096, 2)\n",
        "model_vgg.classifier._modules['7'] = torch.nn.LogSoftmax(dim = 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qAr5tj2NKwfF"
      },
      "source": [
        "PyTorch documentation for [LogSoftmax](https://pytorch.org/docs/stable/nn.html#logsoftmax)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jJ3OenJpCdjp",
        "colab": {}
      },
      "source": [
        "print(model_vgg.classifier)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9ITZFX2MCdju",
        "colab": {}
      },
      "source": [
        "model_vgg = model_vgg.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fU9vWWT2CdkN"
      },
      "source": [
        "## Training fully connected module"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qqp2u3IXCdkO"
      },
      "source": [
        "### Creating loss function and optimizer\n",
        "\n",
        "PyTorch documentation for [NLLLoss](https://pytorch.org/docs/stable/nn.html#nllloss) and the [torch.optim module](https://pytorch.org/docs/stable/optim.html#module-torch.optim)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oP1F4yb8CdkO",
        "colab": {}
      },
      "source": [
        "criterion = nn.NLLLoss()\n",
        "lr = 0.001\n",
        "optimizer_vgg = torch.optim.SGD(model_vgg.classifier[6].parameters(),lr = lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tenuLj67CdkS"
      },
      "source": [
        "### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7nNAUibjCdkS",
        "colab": {}
      },
      "source": [
        "def train_model(model,dataloader,size,epochs=1,optimizer=None):\n",
        "    model.train()\n",
        "    \n",
        "    for epoch in range(epochs):\n",
        "        running_loss = 0.0\n",
        "        running_corrects = 0\n",
        "        for inputs,classes in dataloader:\n",
        "            inputs = inputs.to(device)\n",
        "            classes = classes.to(device)\n",
        "            outputs = model(inputs)\n",
        "            loss = criterion(outputs,classes)           \n",
        "            optimizer = optimizer\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "            running_loss += loss.data.item()\n",
        "            running_corrects += torch.sum(preds == classes.data)\n",
        "        epoch_loss = running_loss / size\n",
        "        epoch_acc = running_corrects.data.item() / size\n",
        "        print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_Jts2jK1CdkV",
        "scrolled": false,
        "colab": {}
      },
      "source": [
        "%%time\n",
        "train_model(model_vgg,loader_train,size=dset_sizes['train'],epochs=2,optimizer=optimizer_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xkMYIX21Hc1k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def test_model(model,dataloader,size):\n",
        "    model.eval()\n",
        "    predictions = np.zeros(size)\n",
        "    all_classes = np.zeros(size)\n",
        "    all_proba = np.zeros((size,2))\n",
        "    i = 0\n",
        "    running_loss = 0.0\n",
        "    running_corrects = 0\n",
        "    for inputs,classes in dataloader:\n",
        "        inputs = inputs.to(device)\n",
        "        classes = classes.to(device)\n",
        "        outputs = model(inputs)\n",
        "        loss = criterion(outputs,classes)           \n",
        "        _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "        running_loss += loss.data.item()\n",
        "        running_corrects += torch.sum(preds == classes.data)\n",
        "        predictions[i:i+len(classes)] = preds.to('cpu').numpy()\n",
        "        all_classes[i:i+len(classes)] = classes.to('cpu').numpy()\n",
        "        all_proba[i:i+len(classes),:] = outputs.data.to('cpu').numpy()\n",
        "        i += len(classes)\n",
        "    epoch_loss = running_loss / size\n",
        "    epoch_acc = running_corrects.data.item() / size\n",
        "    print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))\n",
        "    return predictions, all_proba, all_classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "y3JjYayKCdkW",
        "colab": {}
      },
      "source": [
        "predictions, all_proba, all_classes = test_model(model_vgg,loader_valid,size=dset_sizes['valid'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn6cIwZNCdkY",
        "colab": {}
      },
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(loader_valid))\n",
        "\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Ghkbm7ByCdke",
        "colab": {}
      },
      "source": [
        "outputs = model_vgg(inputs[:n_images].to(device))\n",
        "print(torch.exp(outputs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "F1qdUtvjCdkg",
        "colab": {}
      },
      "source": [
        "classes[:n_images]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0n0hLxTHc1w",
        "colab_type": "text"
      },
      "source": [
        "## Speeding up the learning by precomputing features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "39AALjfgHc1y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_try = model_vgg.features(inputs_try)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bwjaBq8lHc11",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_try.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mZegyLN3Hc13",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preconvfeat(dataloader):\n",
        "    conv_features = []\n",
        "    labels_list = []\n",
        "    for data in dataloader:\n",
        "        inputs,labels = data\n",
        "        inputs = inputs.to(device)\n",
        "        labels = labels.to(device)\n",
        "        \n",
        "        x = model_vgg.features(inputs)\n",
        "        conv_features.extend(x.data.cpu().numpy())\n",
        "        labels_list.extend(labels.data.cpu().numpy())\n",
        "    conv_features = np.concatenate([[feat] for feat in conv_features])\n",
        "    return (conv_features,labels_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UQiycgsqHc1_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "conv_feat_train,labels_train = preconvfeat(loader_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imSeW6-7Hc2D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "conv_feat_train.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q3e5bjsuHc2F",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "conv_feat_valid,labels_valid = preconvfeat(loader_valid)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0fpZByOHc2H",
        "colab_type": "text"
      },
      "source": [
        "### Creating a new data generator\n",
        "\n",
        "We will not load images anymore, so we need to build our own data loader."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hpx-uU90Hc2H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def data_gen(conv_feat,labels,batch_size=64,shuffle=True):\n",
        "    labels = np.array(labels)\n",
        "    if shuffle:\n",
        "        index = np.random.permutation(len(conv_feat))\n",
        "        conv_feat = conv_feat[index]\n",
        "        labels = labels[index]\n",
        "    for idx in range(0,len(conv_feat),batch_size):\n",
        "        yield(conv_feat[idx:idx+batch_size],labels[idx:idx+batch_size])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lK875TWzHc2J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_model(model,size,epochs=1,optimizer=None):\n",
        "    model.train()\n",
        "    \n",
        "    for epoch in range(epochs):\n",
        "        dataloader = data_gen(conv_feat_train,labels_train)\n",
        "        running_loss = 0.0\n",
        "        running_corrects = int(0)\n",
        "        for inputs,classes in dataloader:\n",
        "            inputs , classes = torch.from_numpy(inputs).to(device), torch.from_numpy(classes).to(device)\n",
        "            inputs = inputs.to(device)\n",
        "            classes = classes.to(device)\n",
        "            inputs = inputs.view(inputs.size(0), -1)\n",
        "            outputs = model(inputs)\n",
        "            loss = criterion(outputs,classes)\n",
        "            optimizer = optimizer\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "            running_loss += loss.data.item()\n",
        "            running_corrects += torch.sum(preds == classes.data)\n",
        "        epoch_loss = running_loss / size\n",
        "        epoch_acc = running_corrects.item() / size\n",
        "        print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SFNC9PyGHc2P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "train_model(model_vgg.classifier,size=dset_sizes['train'],epochs=50,optimizer=optimizer_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5N6rbg-Hc2T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def test_model(model,size):\n",
        "    model.eval()\n",
        "    predictions = np.zeros(size)\n",
        "    all_classes = np.zeros(size)\n",
        "    all_proba = np.zeros((size,2))\n",
        "    i = 0\n",
        "    running_loss = 0.0\n",
        "    running_corrects = 0\n",
        "    for inputs,classes in data_gen(conv_feat_valid,labels_valid,shuffle=False):\n",
        "        inputs , classes = torch.from_numpy(inputs).to(device), torch.from_numpy(classes).to(device)\n",
        "        inputs = inputs.to(device)\n",
        "        classes = classes.to(device)\n",
        "        inputs = inputs.view(inputs.size(0), -1)\n",
        "        outputs = model(inputs)\n",
        "        loss = criterion(outputs,classes)           \n",
        "        _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "        running_loss += loss.data.item()\n",
        "        running_corrects += torch.sum(preds == classes.data)\n",
        "        #print(i)\n",
        "        predictions[i:i+len(classes)] = preds.to('cpu').numpy()\n",
        "        all_classes[i:i+len(classes)] = classes.to('cpu').numpy()\n",
        "        all_proba[i:i+len(classes),:] = outputs.data.to('cpu').numpy()\n",
        "            #print(preds.cpu())\n",
        "        i += len(classes)\n",
        "    epoch_loss = running_loss / size\n",
        "    epoch_acc = running_corrects.data.item() / size\n",
        "    print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))\n",
        "    return predictions, all_proba, all_classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2yg770yZHc2W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "predictions, all_proba, all_classes = test_model(model_vgg.classifier,size=dset_sizes['valid'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nlBOnslWHc2Y",
        "colab_type": "text"
      },
      "source": [
        "## 4. Viewing model prediction (qualitative analysis)\n",
        "\n",
        "The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.\n",
        "\n",
        "With our first model we should try to overfit before we start worrying about how to handle that - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques after the 2 weeks break...)\n",
        "\n",
        "\n",
        "As well as looking at the overall metrics, it's also a good idea to look at examples of each of:\n",
        "\n",
        "   1. A few correct labels at random\n",
        "   2. A few incorrect labels at random\n",
        "   3. The most correct labels of each class (ie those with highest probability that are correct)\n",
        "   4. The most incorrect labels of each class (ie those with highest probability that are incorrect)\n",
        "   5. The most uncertain labels (ie those with probability closest to 0.5).\n",
        "\n",
        "In general, these are particularly useful for debugging problems in the model. Since our model is very simple, there may not be too much to learn at this stage..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EJr5EysXHc2a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Number of images to view for each visualization task\n",
        "n_view = 8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_hokobPHc2i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "correct = np.where(predictions==all_classes)[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4Y8kRv-Hc2o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "len(correct)/dset_sizes['valid']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "72LZk1rXHc2v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from numpy.random import random, permutation\n",
        "idx = permutation(correct)[:n_view]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wrel6SmsHc2z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "idx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rQ2pQ7p_Hc22",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "loader_correct = torch.utils.data.DataLoader([dsets['valid'][x] for x in idx],batch_size = n_view,shuffle=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j-Phdh-tHc25",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for data in loader_correct:\n",
        "    inputs_cor,labels_cor = data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3PIV_a2XHc27",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs_cor)\n",
        "\n",
        "imshow(out, title=[l.item() for l in labels_cor])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p8hWSmjJHc2-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from IPython.display import Image, display\n",
        "for x in idx:\n",
        "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hXx3REbpHc3B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "incorrect = np.where(predictions!=all_classes)[0]\n",
        "for x in permutation(incorrect)[:n_view]:\n",
        "    #print(dsets['valid'].imgs[x][1])\n",
        "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kh7QdEVZHc3E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#3. The images we most confident were cats, and are actually cats\n",
        "correct_cats = np.where((predictions==0) & (predictions==all_classes))[0]\n",
        "most_correct_cats = np.argsort(all_proba[correct_cats,1])[:n_view]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ocYE1fvHHc3I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for x in most_correct_cats:\n",
        "    display(Image(filename=dsets['valid'].imgs[correct_cats[x]][0], retina=True))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SmZkwPEsHc3O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#3. The images we most confident were dogs, and are actually dogs\n",
        "correct_dogs = np.where((predictions==1) & (predictions==all_classes))[0]\n",
        "most_correct_dogs = np.argsort(all_proba[correct_dogs,0])[:n_view]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8jUtIj4Hc3R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for x in most_correct_dogs:\n",
        "    display(Image(filename=dsets['valid'].imgs[correct_dogs[x]][0], retina=True))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hjwIo3gbsGMx"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "What did we do in the end? A simple logistic regression! If the connection is unclear, we'll explain it on a much simpler example in the next course. \n",
        "\n",
        "We probably killed a fly with a sledge hammer.\n",
        "\n",
        "![mouche](https://mlelarge.github.io/dataflowr-web/images/mouche.jpg)\n",
        "\n",
        "In our case, the sledge hammer is VGG pretrained on Imagenet, a dataset containing a lot of pictures of cats and dogs. Indeed, we saw that without modification the network was able to predict dog and cat breeds. Hence it is not very surprising that the features computed by VGG are very accurate for our classification task. In the end, we need to learn only the parameters of the last linear layer, i.e. 8194 parameters (do not forget the bias $2\\times 4096+2$). Indeed, this can be done on CPU without any problem.\n",
        "\n",
        "Nevertheless, this example is still instructive as it shows all the necessary steps in a deep learning project. Here we did not struggle with the learning process of a deep network, but we did all the preliminary engineering tasks: dowloading a dataset, setting up the environment to use a GPU, preparing the data, computing the features with a pretrained VGG, saving them on your drive so that you can use them for a later experiment... These steps are essential in any deep learning project and a necessary requirement before having fun playing with network architectures and understanding the learning process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_NhP9XMbuK1A",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hMAwUx7QCdjv"
      },
      "source": [
        "### Saving preconvoluted features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4DQHnbD2H6BJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -U bcolz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3-APvzkACdkD",
        "colab": {}
      },
      "source": [
        "import bcolz\n",
        "\n",
        "def save_array(fname, arr):\n",
        "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
        "    c.flush()\n",
        "def load_array(fname):\n",
        "    return bcolz.open(fname)[:]\n",
        "\n",
        "\n",
        "%mkdir /content/data/dogscats/vgg16\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "AWadFbvtCdkF",
        "colab": {}
      },
      "source": [
        "save_array(os.path.join(data_dir,'vgg16','feat_train.bc'),conv_feat_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_train.bc'),labels_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','feat_val.bc'),conv_feat_valid)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_val.bc'),labels_valid)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pyjlrH4wCdkH"
      },
      "source": [
        "### Uploading Precomputed features\n",
        "\n",
        "This section will allow you to store the precomputed features on your Google drive for later use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "27gRV8KiJw6H",
        "colab": {}
      },
      "source": [
        "%cd /content/data/dogscats/\n",
        "!zip -r vgg16 vgg16/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "de4y59inJx_p",
        "colab": {}
      },
      "source": [
        "!pip install -U -q PyDrive\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "# 1. Authenticate and create the PyDrive client.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JIaZyCpcJ1Hd",
        "colab": {}
      },
      "source": [
        "upload = drive.CreateFile({'title': 'vgg16_drive.zip'})\n",
        "upload.SetContentFile('vgg16.zip')\n",
        "upload.Upload()\n",
        "print('Uploaded file with ID {}'.format(upload.get('id')))"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}